Led/oled array approach to integrated display, focusing lensless light-field camera, and touch-screen user interface devices and associated processors

ABSTRACT

A system for implementing a display which also serves as one or more of a tactile user interface touchscreen, light-field sensor, proximate hand gesture sensor, and focusing lensless light-field imaging camera using image formation systems and methods such as those taught in the inventor&#39;s related patents as cited. In an implementation, an OLED array or LED array that can comprise photosensors that can be used for light sensing as well as light emission functions. The resulting arrangements are advantageous for use in handheld devices such as cellphone, smartphones, PDAs, tablet computers, and other systems and devices.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.13/180,345, filed Jul. 11, 2011, which claims benefit of priority fromU.S. Provisional Application No. 61/363,181, filed Jul. 9, 2010, thecontents of each of which are incorporated herein by reference in theirentireties.

COPYRIGHT & TRADEMARK NOTICES

Certain marks referenced herein may be common law or registeredtrademarks of the applicant, the assignee or third parties affiliated orunaffiliated with the applicant or the assignee. Use of these marks isfor providing an enabling disclosure by way of example and shall not beconstrued to exclusively limit the scope of the disclosed subject matterto material associated with such marks.

BACKGROUND OF THE INVENTION

Throughout the discussion, although “OLED” is in places called outspecifically, an “Organic Light Emitting Diode” (OLED) is regarded as atype of “Light Emitting Diode” (LED). The term “inorganic-LED” is usedto specifically signify traditional LEDs made of non-organic materialssuch as silicon, indium-phosphide, etc. FIG. 1 depicts a visualclassification representation showing inorganic-LEDs and Organic LightEmitting Diodes (OLEDs) as mutually-exclusive types of Light EmittingDiodes (LEDs).

Color inorganic-LED array displays are currently employed in “LED TV”products and road-side and arena color-image LED advertising signs.

Color OLED array displays have begun to appear in cellphones,smartphones, and Personal Digital Assistants (“PDAs”) manufactured bySamsung, Nokia, LG, HTC, Phillips, Sony and others. Color OLED arraydisplays are of particular interest, in general and as pertaining to thepresent invention, because:

-   -   They can be fabricated (along with associated electrical wiring        conductors) via printed electronics on a wide variety of        surfaces such as glass, Mylar, plastics, paper, etc.;    -   Leveraging some such surface materials, they can be readily        bent, printed on curved surfaces, etc.;    -   They can be transparent (and be interconnected with transparent        conductors);    -   Leveraging such transparency, they can be:        -   Stacked vertically,        -   Used as an overlay element atop an LCD or other display,        -   Used as an underlay element between an LCD and its            associated backlight.

LEDs as Light Sensors

Light detection is typically performed by photosite CCD (charge-coupleddevice) elements, phototransistors, CMOS photodetectors, andphotodiodes. Photodiodes are often viewed as the simplest and mostprimitive of these, and typically comprise a PIN(P-type/Intrinstic/N-type) junction rather than the more abrupt PIN(P-type/N-type) junction of conventional signal and rectifying diodes.

However, virtually all diodes are capable of photovoltaic properties tosome extent. In particular, LEDs, which are diodes that have beenstructured and doped specific types of optimized light emission, canalso behave as (at least low-to moderate performance) photodiodes. Inpopular circles Forrest M. Mims has often been credited as callingattention to the fact that that a conventional LED can be used as aphotovoltaic light detector as well as a light emitter (Mims III,Forrest M. “Sun Photometer with Light-emitting diodes as spectrallyselective detectors” Applied Optics, Vol. 31, No. 33, Nov. 20, 1992),and as a photodetector LEDs exhibit spectral selectivity associated withthe LED's emission wavelength. More generally, inorganic-LEDs, organicLEDs (“OLEDs”), organic field effect transistors, and other relateddevices exhibit a range of readily measurable photo-responsiveelectrical properties, such as photocurrents and related photovoltagesand accumulations of charge in the junction capacitance of the LED.

Further, the relation between the spectral detection band and thespectral emission bands of each of a plurality of colors and types ofcolor inorganic-LEDs, OLEDs, and related devices can be used to create acolor light-field sensor from, for example, a color inorganic-LED, OLED,and related device array display. Such arrangements have been describedin pending U.S. patent application Ser. No. 12/419,229 (priority dateJan. 27, 1999), pending U.S. patent application Ser. No. 12/471,275(priority date May 25, 2008), and in pending U.S. patent applicationsSer. Nos. U.S. 13/072,588 and U.S. 61/517,454. The present inventionexpands further upon this.

Pending U.S. patent applications Ser. Nos. 12/419,229 (priority dateJan. 27, 1999), 12/471,275 (priority date May 25, 2008), U.S.13/072,588, and U.S. 61/517,454 additionally teaches how such alight-field sensor can be used together with signal processing softwareto create lensless-imaging camera technology, and how such technologycan be used to create an integrated camera/display device which can beused, for example, to deliver precise eye-contact in video conferencingapplications.

In an embodiment provided for by the invention, each LED in an array ofLEDs can be alternately used as a photodetector or as a light emitter.At any one time, each individual LED would be in one of three states:

-   -   A light emission state,    -   A light detection state,    -   An idle state.

as may be advantageous for various operating strategies. The statetransitions of each LED may be coordinated in a wide variety of ways toafford various multiplexing, signal distribution, and signal gatheringschemes as may be advantageous.

Leveraging this in various ways, in accordance with embodiments of theinvention, array of inorganic-LEDs, OLEDs, or related optoelectronicdevices is configured to perform functions of two or more of:

-   -   a visual image display (graphics, image, video, GUI, etc.),    -   a (lensless imaging) camera,    -   a tactile user interface (touch screen),    -   a proximate gesture user interface.

These arrangements further advantageously allow for a common processorto be used for two or more display, user interface, and camerafunctionalities.

The result dramatically decreases the component count, system hardwarecomplexity, and inter-chip communications complexity for contemporaryand future mobile devices such as cellphones, smartphones, PDAs, tabletcomputers, and other such devices.

In cases where a user interface is incorporated, the RF capacitivematrix arrangements used in contemporary multi-touch touchscreen arereplaced with an optical interface.

Further, the integrated camera/display operation removes the need for ascreen-direction camera and interface electronics in mobile devices. Theintegrated camera/display operation can also improve eye contact inmobile devices.

SUMMARY OF THE INVENTION

For purposes of summarizing, certain aspects, advantages, and novelfeatures are described herein. Not all such advantages may be achievedin accordance with any one particular embodiment. Thus, the disclosedsubject matter may be embodied or carried out in a manner that achievesor optimizes one advantage or group of advantages without achieving alladvantages as may be taught or suggested herein.

In accordance with embodiments of the invention, a system is taught forimplementing a display which also serves as one or more of a tactileuser interface touchscreen, light-field sensor, proximate hand gesturesensor, and lensless imaging camera. In an implementation, an OLED arraycan be used for light sensing as well as light emission functions. Inone implementation a single OLED array is used as the onlyoptoelectronic user interface element in the system. In anotherimplementation two OLED arrays are used, each performing and/oroptimized from different functions. In another implementation, an LCDand an OLED array are used in various configurations. The resultingarrangements allow for sharing of both optoelectric devices as well asassociated electronics and computational processors, and are accordinglyadvantageous for use in handheld devices such as cellphone, smartphones,PDAs, tablet computers, and other such devices.

In accordance with embodiments of the invention, array ofinorganic-LEDs, OLEDs, or related optoelectronic devices is configuredto perform functions of a display, a camera, and a hand-operated userinterface sensor.

In an embodiment, an array of inorganic-LEDs, OLEDs, or related devices,together with associated signal processing aspects of the invention, canbe used to implement a tactile user interface.

In an embodiment, an array of inorganic-LEDs, OLEDs, or related devices,together with associated signal processing aspects of the invention, canbe adapted to function as both an image display and light-field sensorwhich can be used to implement a proximate image user interface.

In an embodiment, an array of inorganic-LEDs, OLEDs, or related devices,together with associated signal processing aspects of the invention, canbe adapted to function as both an image display and light-field sensorwhich can be used to implement a tactile user interface sensor.

In an embodiment, an array of inorganic-LEDs, OLEDs, or related devices,together with associated signal processing aspects of the invention, canbe adapted to function as both an image display and light-field sensorwhich can be used to implement a proximate image user interface sensor.

In an embodiment, an array of inorganic-LEDs, OLEDs, or related devices,together with associated signal processing aspects of the invention, canbe adapted to function as both an image display and light-field sensorwhich can be used to implement a tactile user interface (touch screen),lensless imaging camera, and visual image display.

In an embodiment, an array of inorganic-LEDs, OLEDs, or related devices,together with associated signal processing aspects of the invention, canbe adapted to function as both an image display and light-field sensorwhich can be used to implement a proximate gesture user interface,lensless imaging camera, and visual image display.

In an embodiment, an array of inorganic-LEDs, OLEDs, or related devices,together with associated signal processing aspects of the invention, canbe adapted to function as both an image display and light-field sensorwhich can be used to implement a proximate gesture user interface,tactile user interface (touch screen), lensless imaging camera, andvisual image display.

In accordance with embodiments of the invention, a system forimplementing the function of a visual display, light-field sensor , anda user interface for operated by a user hand is taught, the systemcomprising:

-   -   A processor, the processor having an electrical interface and        for executing at least one software algorithm;    -   A transparent OLED array comprising at least a plurality of        OLEDs, the OLED array configured to be in communication with the        electrical interface of the processor;    -   An optical vignetting arrangement for providing a plurality of        distinct vignettes of an incoming light-field, and    -   A light emitting arrangement associated with the transparent        OLED array, the light emitting arrangement for providing a        visual display;    -   Wherein each distinct vignette of the incoming light-field is        directed to an associated individual OLED from the plurality of        OLEDs;    -   Wherein each of the individual OLED performs a light detection        function at least for an interval of time, the light detection        function comprising a photoelectric effect that is communicated        to the processor via the electrical interface of the processor;        and    -   Wherein the photoelectric effect that is communicated to the        processor is used to obtain light-field measurement information        responsive to the incoming light-field.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features and advantages of the presentinvention will become more apparent upon consideration of the followingdescription of preferred embodiments taken in conjunction with theaccompanying drawing figures, wherein:

FIG. 1 depicts a visual classification representation showinginorganic-LEDs and Organic Light Emitting Diodes (OLEDs) asmutually-exclusive types of Light Emitting Diodes (LEDs).

FIG. 2 depicts a representation of the spread of electron energy levelsas a function of the number of associated electrons in a system such asa lattice of semiconducting material resultant from quantum stateexclusion processes. (The relative positions vertically and fromcolumn-to-column are schematic and not to scale, and electron pairingeffects are not accurately represented.)

FIG. 3 depicts an example electron energy distribution for metals,(wherein the filled valance band overlaps with the conduction band).

FIG. 4 depicts an example electron energy distribution forsemiconductors (wherein the filled valance band is separated from theconduction band by a gap in energy values; this gap is the “band gap”).

FIG. 5 depicts an exemplary (albeit not comprehensive) schematicrepresentation of the relationships between valance bands and conductionbands in materials distinctly classified as metals, semiconductors, andinsulators. (Adapted from Pieter Kuiper,http://en.wikipedia.org/wiki/Electronic_band_structure, visited Mar. 22,2011.)

FIG. 6 depicts the how the energy distribution of electrons in thevalance band and conduction band vary as a function of the density ofelectron states, and the resultant growth of the band gap as the densityof electron states increases. (Adapted from Pieter Kuiper,http://en.wikipedia.org/wiki/Band_gap, visited Mar. 22, 2011.)

FIG. 7 depicts three exemplary types of electron-hole creation processesresulting from absorbed photons that contribute to current flow in a PNdiode (adapted from A. Yariv, Optical Electronics, 4th edition, SaundersCollege Press, 1991, p. 423).

FIG. 8 depicts exemplary electron energy distribution among bonding andantibonding molecular orbitals in conjugated or aromatic organiccompounds (adapted from Y. Divayana, X. Sung, Electroluminescence inOrganic Light-Emitting Diodes, VDM Verlag Dr. Willer, Saarbrücken, 2009,ISBN 978-3-639-17790-9, FIG. 2.2, p.13).

FIG. 9 depicts an optimization space for semiconductor diodes comprisingattributes of signal switching performance, light emitting performance,and light detection performance.

FIG. 10 depicts an exemplary metric space of device realizations foroptoelectronic devices and regions of optimization and co-optimization.

FIGS. 11-14 depict various exemplary circuits demonstrating variousexemplary approaches to detecting light with an LED.

FIG. 15 depicts a selectable grounding capability for a two-dimensionalarray of LEDs.

FIG. 16 depicts an adaptation of the arrangement depicted in FIG. 15that is controlled by an address decoder so that the selected subset canbe associated with a unique binary address.

FIG. 17 depicts an exemplary highly-scalable electrically-multiplexedLED array display that also functions as a light-field detector.

FIGS. 18 and 19 depict exemplary functional cells that may be used in alarge scale array.

FIGS. 20-22 depict adaptations of the digital circuit measurement anddisplay arrangements into an example combination.

FIGS. 23-26 depict exemplary state diagrams for the operation of the LEDand the use of input signals and output signals.

FIG. 27 depicts an exemplary high-level structure of a three-colortransparent Stacked OLED (“SOLED”) element as can be used for colorlight emission, color light sensing, or both. (Adapted from G. Gu, G.Parthasarathy, P. Burrows, T. Tian, I. Hill, A. Kahn, S. Forrest,“Transparent stacked organic light emitting devices. I. Designprinciples and transparent compound electrodes,” Journal of AppliedPhysics, October 1999, vol. 86 no. 8, pp. 4067-4075.)

FIG. 28 depicts a conventional “RGB stripe” OLED array as used in avariety of OLED display products and as can be used for light-fieldsensing in accordance with aspects of the present invention. (Adaptedfromhttp://www.displayblog.com/2009/03/26/samsung-oled-pentile-matrix-next-iphone-oled-display/(visitedMar. 23, 2011.)

FIG. 29 depicts a representation of the “PenTile Matrix” OLED array asattributed to Samsung and as can be used for light-field sensing inaccordance with aspects of the present invention. (Adapted fromhttp://www.displayblog.com/2009/03/26/samsung-oled-pentile-matrix-next-iphone-oled-display/(visitedMar. 23, 2011.)

FIG. 30 depicts an exemplary embodiment comprising an LED array precededby a vignetting arrangement as is useful for implementing a lenslessimaging camera as taught in U.S. patent application Ser. Nos. 12/419,229(priority date Jan. 27, 1999), 12/471,275 (priority date May 25, 2008),U.S. 13/072,588, and U.S. 61/517,454.

FIG. 31 depicts an exemplary implementation comprising a transparentOLED array overlaid upon an LCD display, which is in turn overlaid on a(typically) LED backlight used to create and direct light though the LCDdisplay from behind.

FIG. 32 depicts an exemplary implementation comprising a firsttransparent OLED array used for at least optical sensing overlaid upon asecond OLED array used for at least visual display.

FIG. 33 depicts an exemplary implementation comprising a firsttransparent OLED array used for at least visual display overlaid upon asecond OLED array used for at least optical sensing.

FIG. 34 depicts an exemplary implementation comprising an LCD display,used for at least visual display and vignette formation, overlaid upon asecond LED array, used for at least backlighting of the LCD and opticalsensing.

FIG. 35 an exemplary arrangement employed in contemporary cellphones,smartphones, PDAs, tablet computers, and other portable devices whereina transparent capacitive matrix proximity sensor is overlaid over an LCDdisplay, which is in turn overlaid on a (typically LED) backlight usedto create and direct light though the LCD display from behind. Each ofthe capacitive matrix and the LCD have considerable associatedelectronic circuitry and software associated with them.

FIG. 36 depicts an exemplary modification of the arrangement depicted inFIG. 35 wherein the LCD display and backlight are replaced with an OLEDarray used as a visual display. Such an arrangement has started to beincorporated in recent contemporary cellphone, smartphone, PDA, tabletcomputers, and other portable device products by several manufacturers.

FIG. 37 depicts an exemplary arrangement provided for by the inventioncomprising only a LED array. The LEDs in the LED array may be OLEDs orinorganic LEDs. Such an arrangement can be used as a visual display andas a tactile user interface.

FIG. 38 depicts a representative exemplary arrangement wherein lightemitted by neighboring LEDs is reflected from a finger back to an LEDacting as a light sensor.

FIG. 39 depicts an exemplary arrangement wherein a particular LEDdesignated to act as a light sensor is surrounded byimmediately-neighboring LEDs designated to emit light to illuminate thefinger for example as depicted in FIG. 38.

FIG. 40 depicts an exemplary arrangement wherein a particular LEDdesignated to act as a light sensor is surrounded byimmediately-neighboring LEDs designated to serve as a “guard” area, forexample not emitting light, these in turn surrounded byimmediately-neighboring LEDs designated to emit light used to illuminatethe finger for example as depicted in FIG. 38.

FIG. 41 depicts an exemplary reference arrangement comprised by mobiledevices such as cellphones, smartphones, PDAs, tablet computers, andother such devices.

FIG. 42 depicts an exemplary variation of the exemplary referencearrangement of FIG. 41 wherein an LED array replaces the display,camera, and touch sensor and is interfaced by a common processor thatreplaces associated support hardware.

FIG. 43 depicts an exemplary variation of the exemplary referencearrangement of FIG. 42 wherein the common processor associated with theLED array further executes at least some touch-based user interfacesoftware.

FIG. 44 depicts an exemplary variation of the exemplary referencearrangement of FIG. 42 wherein the common processor associated with theLED array further executes all touch-based user interface software.

FIG. 45 depicts an exemplary embodiment comprising an LED array precededby a micro optics array and a capacitive matrix sensor

FIG. 46 depicts an exemplary embodiment comprising an LED array precededby a micro optics array configured to also function as a capacitivematrix sensor.

DETAILED DESCRIPTION

The above and other aspects, features and advantages of the presentinvention will become more apparent upon consideration of the followingdescription of preferred embodiments taken in conjunction with theaccompanying drawing figures.

In the following description, reference is made to the accompanyingdrawing figures which form a part hereof, and which show by way ofillustration specific embodiments of the invention. It is to beunderstood by those of ordinary skill in this technological field thatother embodiments may be utilized, and structural, electrical, as wellas procedural changes may be made without departing from the scope ofthe present invention.

Inorganic and Organic Semiconductors

FIG. 2 depicts a representation of the spread of electron energy levelsas a function of the number of associated electrons in a system such asa lattice of semiconducting material resultant from quantum stateexclusion processes. As the number of associated electrons in a systemincreases, the separation between consecutive energy levels decreases,in the limit becoming an effective continuum of energy levels. Higherenergy level electrons form a conduction band while lower energyelectrons lie in a valence band. The relative positions vertically andfrom column-to-column are schematic and not to scale, and electronpairing effects are not accurately represented.

FIG. 3 depicts an example electron energy distribution for metals,(wherein the filled valance band overlaps with the conduction band).

FIG. 4 depicts an example electron energy distribution forsemiconductors; here the filled valance band is separated from theconduction band by a gap in energy values. The “band gap” is thedifference in energy between electrons at the top of the valence bandand electrons at the bottom of the conduction band. For a semiconductorthe band gap is small, and manipulations of materials, physicalconfigurations, charge and potential differences, photon absorption,etc. can be used to move electrons through the band gap or along theconduction band.

Elaborating further, FIG. 5 (adapted from Pieter Kuiper,http://en.wikipedia.org/wiki/Electronic_band_structure, visited Mar. 22,2011) depicts an exemplary (albeit not comprehensive) schematicrepresentation of the relationships between valance bands and conductionbands in materials distinctly classified as metals, semiconductors, andinsulators. The band gap is a major factor determining the electricalconductivity of a material. Although metal conductor materials are shownhaving overlapping valance and conduction bands, there are someconductors that instead have very small band gaps. Materials withsomewhat larger band gaps are electrical semiconductors, while materialswith very large band gaps are electrical insulators.

The affairs shown in FIG. 1 and FIG. 4 are related. FIG. 6 (adapted fromPieter Kuiper, http://en.wikipedia.org/wiki/Band_gap, visited Mar. 22,2011) depicts how the energy distribution of electrons in the valanceband and conduction band vary as a function of the density of assumedelectron states per unit of energy, illustrating growth of the size ofthe band gap as the density of states (horizontal axis) increases.

Light Sensing by Photodiodes and LEDs

Electrons can move between the valence band and the conduction band bymeans of special processes that give rise to hole-electron generationand hole-electron recombination. Several such processes are related tothe absorption and emission of photons which make up light.

Light detection in information systems is typically performed byphotosite CCD (charge-coupled device) elements, phototransistors, CMOSphotodetectors, and photodiodes. By way of example, FIG. 7 (adapted fromA. Yariv, Optical Electronics, 4th edition, Saunders College Press,1991, p. 423) depicts three exemplary types of electron-hole creationprocesses resulting from absorbed photons that contribute to currentflow in a PN diode. Emitted photons cause electrons to drop through theband gap while absorbed photons of sufficient energy can exciteelectrons from the valance band though the band gap to the conductionband.

Photodiodes are often viewed as the simplest and most primitive form ofsemiconductor light detector. A photodiode typically comprises a PIN(P-type/Intrinsic/N-type) junction rather than the more abrupt PIN(P-type/N-type) junction of conventional signal and rectifying diodes.However, photoelectric effects and capabilities are hardly restricted toPIN diode structures. In varying degrees, virtually all diodes arecapable of photovoltaic properties to some extent.

In particular, LEDs, which are diodes that have been structured anddoped for specific types of optimized light emission, can also behave as(at least low-to-medium performance) photodiodes. Additionally, LEDsalso exhibit other readily measurable photo-responsive electricalproperties, such as photodiode-type photocurrents and relatedaccumulations of charge in the junction capacitance of the LED. Inpopular circles Forrest M. Mims has often been credited as callingattention to the fact that that a conventional LED can be used as aphotovoltaic light detector as well as a light emitter (Mims III,Forrest M. “Sun Photometer with Light-emitting diodes as spectrallyselective detectors” Applied Optics, Vol. 31, No. 33, Nov. 20, 1992).More generally LEDs, organic LEDs (“OLEDs”), organic field effecttransistors, and other related devices exhibit a range of readilymeasurable photo-responsive electrical properties, such as photocurrentsand related photovoltages and accumulations of charge in the junctioncapacitance of the LED.

In an LED, light is emitted when holes and carriers recombine and thephotons emitted have an energy in a small range either side of theenergy span of the band gap. Through engineering of the band gap, thewavelength of light emitted by an LED can be controlled. In theaforementioned article, Mims additionally pointed out that as aphotodetector LEDs exhibit spectral selectivity with at a lightabsorption wavelength similar to that of the LED's emission wavelength.More details as to the spectral selectivity of the photoelectricresponse of an LED will be provided later.

Attention is now directed to organic semiconductors and their electricaland optoelectrical behavior. Conjugated organic compounds comprisealternating single and double bonds in the local molecular topologycomprising at least some individual atoms (usually carbon, but can beother types of atoms) in the molecule. The resulting electric fieldsorganize the orbitals of those atoms into a hybrid formation comprisinga σ bond (which engage electrons in forming the molecular structureamong joined molecules) and a π cloud of loosely associated electronsthat are in fact delocalized and can move more freely within themolecule. These delocalized π electrons provide a means for chargetransport within the molecule and electric current withinlarger-structures of organic materials (for example, polymers).

Combinations of atomic orbital modalities for the individual atoms in amolecule, together with the molecular topology (created by the σ bonds)and molecular geometry, create molecule-scale orbitals for thedelocalized π cloud of electrons and in a sense for the electronscomprising σ bonds. Interactions among the electrons, in particularquantum exclusion processes, create an energy gap between the HighestOccupied Molecular Orbital (“HOMO”) and Lowest-Unoccupied MolecularOrbital (“LUMO”) for the delocalized π electrons (and similarly does sofor the more highly localized σ bond electrons). FIG. 8 (adapted from Y.Divayana, X. Sung, Electroluminescence in Organic Light-Emitting Diodes,VDM Verlag Dr. Willer, SaarbrUcken, 2009, ISBN 978-3-639-17790-9, FIG.2.2, p. 13) depicts the electron energy distribution among bonding (πand σ) and antibonding (π* and σ*) molecular orbitals in for twoelectrons in an exemplary conjugated or aromatic organic compound. Insuch materials typically the energy gap between the π and π* molecularorbitals correspond to the gap between the HOMO and LUMO. The HOMOeffectively acts as a valence band in a traditional (inorganic) crystallattice semiconductor and the LUMO acts as effective equivalent to aconduction band. Accordingly, energy gap between the HOMO and LUMO(usually corresponding to the gap between the π and π* molecularorbitals) behaves in a manner similar to the band gap in a crystallattice semiconductor and thus permits many aromatic organic compoundsto serve as electrical semiconductors.

Emitted photons cause electrons to drop through the HOMO/LUMO gap whileabsorbed photons of sufficient energy can excite electrons from the HOMOto the LUMO. These processes are similar to photon emission and photoabsorption processes in a crystal lattice semiconductor and can be usedto implement organic LED (“OLED”) and organic photodiode effects witharomatic organic compounds. Functional groups and other factors can varythe width of the band gap so that it matches energy transitionsassociated with selected colors of visual light. Additional details onorganic LED (“OLED”) processes, materials, operation, fabrication,performance, and applications can be found in, for example:

-   -   Z. Li, H. Ming (eds.), Organic Light-Emitting Materials and        Devices, CRC Taylor & Francis, Boca Raton, 2007, ISBN        1-57444-574-X;    -   Z. Kafafi (ed.), Organic Electroluminescence, CRC Taylor &        Francis, Boca Raton, 2005, ISBN 0-8247-5906-0;    -   Y. Divayana, X. Sung, Electroluminescence in Organic        Light-Emitting Diodes, VDM Verlag Dr. Willer, Saarbrücken, 2009,        ISBN 978-3-639-17790-9.

It is noted that an emerging alternative to OLEDs are Organic LightEmitting Transistors (OLETS). The present invention allows forarrangements employing OLETS to be employed in place of OLEDs and LEDsas appropriate and advantageous wherever mentioned throughout thespecification.

Potential Co-optimization of Light Sensing and Light EmittingCapabilities of an Optical Diode Element

FIG. 9 depicts an optimization space 200 for semiconductor (traditionalcrystal lattice or organic material) diodes comprising attributes ofsignal switching performance, light emitting performance, and lightdetection performance. Specific diode materials, diode structure, anddiode fabrication approaches 223 can be adjusted to optimize a resultantdiode for switching function performance 201 (for example, via use ofabrupt junctions), light detection performance 202 (for example via aP-I-N structure comprising a layer of intrinsic semiconducting materialbetween regions of n-type and p-type material, or light detectionperformance 203.

FIG. 10 depicts an exemplary metric space 1900 of device realizationsfor optoelectronic devices and regions of optimization andco-optimization.

Specific optoelectrical diode materials, structure, and fabricationapproaches 1923 can be adjusted to optimize a resultant optoelectricaldiode for light detection performance 1901 (for example via a P-I-Nstructure comprising a layer of intrinsic semiconducting materialbetween regions of n-type and p-type material versus light emissionperformance 1902 versus cost 1903. Optimization within the plane definedby light detection performance 1901 and cost 1903 traditionally resultin photodiodes 1911 while optimization within the plane defined by lightemission performance 1902 and cost 1903 traditionally result in LEDs1912. The present invention provides for specific optoelectrical diodematerials, structure, and fabrication approaches 1923 to be adjusted toco-optimize an optoelectrical diode for both good light detectionperformance 1901 and light emission performance 1902 versus cost 1903. Aresulting co-optimized optoelectrical diode can be used for multiplexedlight emission and light detection modes. These permit a number ofapplications as explained in the sections to follow.

Again it is noted that an emerging alternative to OLEDs are OrganicLight Emitting Transistors (OLETS). The present invention allows forarrangements employing OLETS to be employed in place of OLEDs and LEDsas appropriate and advantageous wherever mentioned throughout thespecification.

Electronic Circuit Interfacing to LEDs Used as Light Sensors

FIGS. 11-14 depict various exemplary circuits demonstrating variousexemplary approaches to detecting light with an LED. These initiallyintroduce the concepts of received light intensity measurement(“detection”) and varying light emission intensity of an LED in terms ofvariations in D.C. (“direct-current”) voltages and currents. However,light intensity measurement (“detection”) can be accomplished by othermeans such as LED capacitance effects—for example reverse-biasing theLED to deposit a known charge, removing the reverse bias, and thenmeasuring the time for the charge to then dissipate within the LED.Also, varying the light emission intensity of an LED can be accomplishedby other means such as pulse-width-modulation—for example, a duty-cycleof 50% yields 50% of the “constant-on” brightness, a duty-cycle of 50%yields 50% of the “constant-on” brightness, etc. These, too, areprovided for by the invention and will be considered again later asvariations of the illustrative approaches provided below.

To begin, LED1 in FIG. 11 is employed as a photodiode, generating avoltage with respect to ground responsive to the intensity of the lightreceived at the optically-exposed portion of the LED-structuredsemiconducting material. In particular, for at least a range of lightintensity levels the voltage generated by LED1 increases monotonicallywith the received light intensity. This voltage can be amplified by ahigh-impedance amplifier, preferably with low offset currents. Theexample of FIG. 11 shows this amplification performed by a simpleoperational amplifier (“op amp”) circuit with fractional negativefeedback, the fraction determined via a voltage divider. The gainprovided by this simple op amp arrangement can be readily recognized byone skilled in the art as

1+(R _(f) /R _(g)).

The op amp produces an isolated and amplified output voltage thatincreases, at least for a range, monotonically with increasing lightreceived at the light detection LED 1. Further in this exampleillustrative circuit, the output voltage of the op amp is directed toLED100 via current-limiting resistor R100. The result is that thebrightness of light emitted by LED100 varies with the level of lightreceived by LED1.

For a simple lab demonstration of this rather remarkable fact, one canchoose a TL08x series (TL082, TL084, etc.) or equivalent op amp poweredby +12 and −12 volt split power supply, R100 of ˜1 LΩ, and R_(f)/R_(g)in a ratio ranging from 1 to 20 depending on the type of LED chosen.LED100 will be dark when LED1 is engulfed in darkness and will bebrightly lit when LED1 is exposed to natural levels of ambient roomlight. For best measurement studies, LED1 could comprise a “water-clear”plastic housing (rather than color-tinted). It should also be noted thatthe LED1 connection to the amplifier input is of relatively quite highimpedance and as such can readily pick up AC fields, radio signals, etc.and is best realized using as physically small electrical surface areaand length as possible. In a robust system, electromagnetic shielding isadvantageous.

The demonstration circuit of FIG. 11 can be improved, modified, andadapted in various ways (for example, by adding voltage and/or currentoffsets, JFET preamplifiers, etc.), but as shown is sufficient to showthat a wide range of conventional LEDs can serve as pixel sensors for anambient-room light sensor array as can be used in a camera or otherroom-light imaging system. Additionally, LED100 shows the role an LEDcan play as a pixel emitter of light.

FIG. 12 shows a demonstration circuit for the photocurrent of the LED.For at least a range of light intensity levels the photocurrentgenerated by LED1 increases monotonically with the received lightintensity. In this exemplary circuit the photocurrent is directed to anatively high-impedance op amp (for example, a FET input op amp such asthe relatively well-known LF-351) set up as an invertingcurrent-to-voltage converter. The magnitude of the transresistance(i.e., the current-to-voltage “gain”) of this invertingcurrent-to-voltage converter is set by the value of the feedbackresistor Rf. The resultant circuit operates in a similar fashion to thatof FIG. 11 in that the output voltage of the op amp increases, at leastfor a range, monotonically with increasing light received at the lightdetection LED. The inverting current-to-voltage converter inverts thesign of the voltage, and such inversion in sign can be corrected by alater amplification stage, used directly, or is preferred. In othersituations it can be advantageous to not have the sign inversion, inwhich case the LED orientation in the circuit can be reversed, as shownin FIG. 13.

FIG. 14 shows an illustrative demonstration arrangement in which an LEDcan be for a very short duration of time reverse biased and then in asubsequent interval of time the resultant accumulations of charge in thejunction capacitance of the LED are discharged. The decrease in chargeduring discharge through the resistor R results in a voltage that can bemeasured with respect to a predetermined voltage threshold, for exampleas can be provided by a (non-hysteretic) comparator or (hysteretic)Schmitt-trigger. The resulting variation in discharge time variesmonotonically with the light received by the LED. The illustrativedemonstration arrangement provided in FIG. 14 is further shown in thecontext of connects to the bidirectional I/O pin circuit for aconventional microprocessor. This permits the principal to be readilydemonstrated through a simple software program operating on such amicroprocessor. Additionally, as will be seen later, the very samecircuit arrangement can be used to variably control the emitted lightbrightness of the LED by modulating the temporal pulse-width of a binarysignal at one or both of the microprocessor pins.

Multiplexing Circuitry for LED Arrays

For rectangular arrays of LEDs, it is typically useful to interconnecteach LED with access wiring arranged to be part of a correspondingmatrix wiring arrangement. The matrix wiring arrangement istime-division multiplexed. Such time-division multiplexed arrangementscan be used for delivering voltages and currents to selectivelyilluminate each individual LED at a specific intensity level (includingvery low or zero values so as to not illuminate).

An example multiplexing arrangement for a two-dimensional array of LEDsis depicted in FIG. 15. Here each of a plurality of normally-open analogswitches are sequentially closed for brief disjointed intervals of time.This allows the selection of a particular subset (here, a column) ofLEDs to be grounded while leaving all other LEDs in the array notconnected to ground. Each of the horizontal lines then can be used toconnect to exactly one grounded LED at a time. The plurality ofnormally-open analog switches in FIG. 15 may be controlled by an addressdecoder so that the selected subset can be associated with a uniquebinary address, as suggested in FIG. 16. The combination of theplurality of normally-open analog switches together with the addressdecoder form an analog line selector. By connecting the line decoder'saddress decoder input to a counter, the columns of the LED array can besequentially scanned.

FIG. 17 depicts an exemplary adaptation of the arrangement of FIG. 16together to form a highly scalable LED array display that also functionsas a light-field detector. The various multiplexing switches in thisarrangement can be synchronized with the line selector and mode controlsignal so that each LED very briefly provides periodically updateddetection measurement and is free to emit light the rest of the time. Awide range of variations and other possible implementations are possibleand implemented in various products.

Such time-division multiplexed arrangements can alternatively be usedfor selectively measuring voltages or currents of each individual LED.Further, the illumination and measurement time-division multiplexedarrangements themselves can be time-division multiplexed, interleaved,or merged in various ways. As an illustrative example, the arrangementof FIG. 17 can be reorganized so that the LED, mode control switch,capacitor, and amplifiers are collocated, for example as in theillustrative exemplary arrangement of FIG. 18. Such an arrangement canbe implemented with, for example, three MOSFET switching transistorconfigurations, two MOSFET amplifying transistor configurations, asmall-area/small-volume capacitor, and an LED element (that is, fivetransistors, a small capacitor, and an LED). This can be treated as acell which is interconnected to multiplexing switches and control logic.A wide range of variations and other possible implementations arepossible and the example of FIG. 17 is in no way limiting. For example,the arrangement of FIG. 17 can be reorganized to decentralize themultiplexing structures so that the LED, mode control switch,multiplexing and sample/hold switches, capacitor, and amplifiers arecollocated, for example as in the illustrative exemplary arrangement ofFIG. 19. Such an arrangement can be implemented with, for example, threeMOSFET switching transistor configurations, two MOSFET amplifyingtransistor configurations, a small-area/small-volume capacitor, and anLED element (that is, five transistors, a small capacitor, and an LED).This can be treated as a cell whose analog signals are directlyinterconnected to busses. Other arrangements are also possible.

The discussion and development thus far are based on the analog circuitmeasurement and display arrangement of FIG. 11 that in turn leveragesthe photovoltaic properties of LEDs. With minor modifications clear toone skilled in the art, the discussion and development thus far can bemodified to operate based on the analog circuit measurement and displayarrangements of FIG. 12 and FIG. 13 that leverage the photocurrrentproperties of LEDs.

FIG. 20, FIG. 21, and FIG. 22 depict an example of how the digitalcircuit measurement and display arrangement of FIG. 14 (that in turnleverages discharge times for accumulations of photo-induced charge inthe junction capacitance of the LED) can be adapted into theconstruction developed thus far. FIG. 20 adapts FIG. 14 to additionalinclude provisions for illuminating the LED with a pulse-modulatedemission signal. Noting that the detection process described earlier inconjunction with FIG. 14 can be confined to unperceivably shortintervals of time, FIG. 21 illustrates how a pulse-width modulatedbinary signal may be generated during LED illumination intervals to varyLED emitted light brightness. FIG. 22 illustrates an adaptation of thetri-state and Schmitt-trigger/comparator logic akin to that illustratedin the microprocessor I/O pin interface that may be used to sequentiallyaccess subsets of LEDs in an LED array as described in conjunction withFIG. 15 and FIG. 16.

FIGS. 23-25 depict exemplary state diagrams for the operation of the LEDand the use of input signals and output signals described above. Fromthe viewpoint of the binary mode control signal there are only twostates: a detection state and an emission state, as suggested in FIG.23. From the viewpoint of the role of the LED in a larger systemincorporating a multiplexed circuit arrangement such as that of FIG. 17,there may a detection state, an emission state, and an idle state (wherethere is no emission nor detection occurring), obeying state transitionmaps such as depicted in FIG. 24 or FIG. 25. At a further level ofdetail, there are additional considerations:

-   -   To emit light, a binary mode control signal can be set to “emit”        mode (causing the analog switch to be closed) and the emission        light signal must be of sufficient value to cause the LED to        emit light (for example, so that the voltage across the LED is        above the “turn-on” voltage for that LED).        -   If the binary mode control signal is in “emit” mode but the            emission light signal is not of such sufficient value, the            LED will not illuminate. This can be useful for brightness            control (via pulse-width modulation), black-screen display,            and other uses. In some embodiments, this may be used to            coordinate the light emission of neighboring LEDs in an            array while a particular LED in the array is in detection            mode.        -   If the emission light signal of such sufficient value but            the binary mode control signal is in “detect” mode, the LED            will not illuminate responsive to the emission light signal.            This allows the emission light signal to be varied during a            time interval when there is no light emitted, a property            useful for multiplexing arrangements.    -   During a time interval beginning with the change of state of the        binary mode control signal to some settling-time period        afterwards, the detection output and/or light emission level may        momentarily not be accurate.    -   To detect light, the binary mode control signal must be in        “detect” mode (causing the analog switch to be open). The        detected light signal may be used by a subsequent system or        ignored. Intervals where the circuit is in detection mode but        the detection signal is ignored may be useful for multiplexing        arrangement, in providing guard-intervals for settling time, to        coordinate with the light emission of neighboring LEDs in an        array, etc.

FIG. 26 depicts an exemplary state transition diagram reflecting theabove considerations. The top “Emit Mode” box and bottom “Detect Mode”box reflect the states of an LED from the viewpoint of the binary modecontrol signal as suggested by FIG. 23. The two “Idle” states (one ineach of the “Emit Mode” box and “Detect Mode” box) of FIG. 26 reflect(at least in part) the “Idle” state suggested in FIG. 24 and/or FIG. 25.Within the “Emit Mode” box, transitions between “Emit” and “Idle” may becontrolled by emit signal multiplexing arrangements, algorithms forcoordinating the light emission of an LED in an array while aneighboring LED in the array is in detection mode, etc. Within the“Detect Mode” box, transitions between “Detect” and “Idle” may becontrolled by independent or coordinated multiplexing arrangements,algorithms for coordinating the light emission of an LED in an arraywhile a neighboring LED in the array is in detection mode, etc. Inmaking transitions between states in the boxes, the originating andtermination states may be chosen in a manner advantageous for details ofvarious multiplexing and feature embodiments. Transitions between thegroups of states within the two boxes correspond to the vast impedanceshift invoked by the switch opening and closing as driven by the binarymode control signal. In FIG. 26, the settling times between these twogroups of states are gathered and regarded as a transitional state.

As mentioned earlier, the amplitude of light emitted by an LED can bemodulated to lesser values by means of pulse-width modulation (PWM) of abinary waveform. For example, if the binary waveform oscillates betweenfully illuminated and non-illuminated values, the LED illuminationamplitude will be perceived roughly as 50% of the full-on illuminationlevel when the duty-cycle of the pulse is 50%, roughly as 75% of thefull-on illumination level when the duty-cycle of the pulse is 75%,roughly as 10% of the full-on illumination level when the duty-cycle ofthe pulse is 10%, etc. Clearly the larger fraction of time the LED isilluminated (i.e., the larger the duty-cycle), the brighter theperceived light observed emitted from the LED.

Stacked OLEDs (“SOLED”) as Optical Diode Elements for Use in theInvention

FIG. 27 (adapted from G. Gu, G. Parthasarathy, P. Burrows, T. Tian, I.Hill, A. Kahn, S. Forrest, “Transparent stacked organic light emittingdevices. I. Design principles and transparent compound electrodes,”Journal of Applied Physics, October 1999, vol. 86 no. 8, pp.4067-4075)depicts an exemplary high-level structure of a three-color transparentStacked OLED (“SOLED”) element as has been developed for use inlight-emitting color displays.

The present invention provides for a three-color transparent SOLEDelement such as those depicted in FIG. 27 or of other forms developedfor use in light-emitting color displays to be used as-is as a colortransparent light sensor.

Alternatively, the present invention provides for analogous structuresto be used to implement a three-color transparent light sensor, forexample, with reference to FIG. 10, by replacing optoelectrical diodematerials, structure, and fabrication approaches 1923 optimized forlight emission performance 1902 with optoelectrical diode materials,structure, and fabrication approaches 1923 optimized for light detectionperformance 1901. The present invention additionally provides for athree-color transparent SOLED element such as those depicted in FIG. 27or of other forms developed for use in light-emitting color displays toemploy specific optoelectrical diode materials, structure, andfabrication approaches 1923 to be adjusted to co-optimize anoptoelectrical diode for both good light detection performance 1901 andlight emission performance 1902 versus cost 1903. The resultingstructure can be used for color light emission, color light sensing, orboth.

The invention additionally provides for the alternative use of similaror related structures employing OLETs.

Arrayed OLEDs as Optical Diode Elements for Use in the Invention

FIG. 28 (adapted fromhttp://www.displayblog.com/2009/03/26/samsung-oled-pentile-matrix-next-iphone-oled-display/(visitedMar. 23, 2011) depicts a conventional “RGB stripe” OLED array as used ina variety of OLED display products. FIG. 29 (also adapted fromhttp://www.displayblog.com/2009/03/26/samsung-oled-pentile-matrix-next-iphone-oled-display/(visitedMar. 23, 2011) depicts a representation of the “PenTile Matrix” OLEDarray as attributed to Samsung which provides good display performancewith a 33% reduction in pixel element count.

The present invention provides for arrays of OLED elements such as thosedepicted in FIG. 28 and FIG. 29 other forms developed for use inlight-emitting color displays to be used as-is as a color light sensors.This permits OLED elements in arrays such as those depicted in FIG. 28and FIG. 29 or of other forms developed for use in light-emitting colordisplays to be used for light-field sensing in accordance with aspectsof the present invention.

Alternatively, the present invention provides for analogous structuresto be used to implement a three-color transparent light sensor, forexample, with reference to FIG. 10, by replacing optoelectrical diodematerials, structure, and fabrication approaches 1923 optimized forlight emission performance 1902 with optoelectrical diode materials,structure, and fabrication approaches 1923 optimized for light detectionperformance 1901. The present invention additionally provides for OLEDelements in arrays such as those depicted in FIG. 27 or of other formsdeveloped for use in light-emitting color displays to employ specificoptoelectrical diode materials, structure, and fabrication approaches1923 to be adjusted to co-optimize an optoelectrical diode for both goodlight detection performance 1901 and light emission performance 1902versus cost 1903. The resulting structure can be used for color lightemission, color light sensing, or both.

The invention additionally provides for the alternative use of similaror related structures employing OLETs.

Light-Field Sensor Embodiments

In an embodiment, various materials, physical processes, structures, andfabrication techniques used in creating an array of color inorganicLEDs, OLEDs, OLETs, or related devices and associated co-locatedelectronics (such as FETs, resistors, and capacitors) can be used as-isto create an array of color inorganic LEDs, OLEDs, OLETs, or relateddevices well-suited as a color light-field sensor. An exemplary generalframework underlying such an arrangement is described in pending U.S.patent application Ser. No. 12/419,229 (priority date Jan. 27, 1999),pending U.S. patent application Ser. No. 12/471,275 (priority date May25, 2008), and in pending U.S. patent application Ser. Nos. 13/072,588and U.S. 61/517,454.

In an embodiment, various materials, physical processes, structures, andfabrication techniques used in creating an array of color inorganicLEDs, OLEDs, OLETs, or related devices and associated co-locatedelectronics (such as FETs, resistors, and capacitors) can beco-optimized to create an array of color inorganic LEDs, OLEDs, OLETs,or related devices well-suited as a color light-field sensor. Anexemplary general framework underlying such an arrangement is describedin pending U.S. patent application Ser. Nos. 12/419,229 (priority dateJan. 27, 1999), pending U.S. patent application Ser. No. 12/471,275(priority date May 25, 2008), and in pending U.S. patent applicationSer. Nos. U.S. 13/072,588 and U.S. 61/517,454.

In an embodiment, at least three inorganic LEDs, OLEDs, or relateddevices are transparent. In an embodiment, the at least threetransparent inorganic LEDs, OLEDs, or related devices are comprised inan array that is overlaid on a display such as an LCD.

Operation as a Combination Light-field Sensor and Display

In embodiments provided for by the invention, each inorganic LED, OLED,or related device in an array of color inorganic LEDs, OLEDs, OLETs, orrelated devices can be alternately used as a photodetector or as a lightemitter. The state transitions of each inorganic LED, OLED, or relateddevice in the array of color inorganic LEDs, OLEDs, OLETs, or relateddevices among the above states can be coordinated in a wide variety ofways to afford various multiplexing, signal distribution, and signalgathering schemes as can be advantageous. An exemplary general frameworkunderlying such an arrangement is described in pending U.S. patentapplication Ser. No. 12/419,229 (priority date Jan. 27, 1999), pendingU.S. patent application Ser. No. 12/471,275 (priority date May 25,2008), and in pending U.S. patent application Ser. Nos. U.S. 13/072,588and U.S. 61/517,454.

In embodiments provided for by the invention, each inorganic LED, OLED,or related device in an array of inorganic LEDs, OLEDs, or relateddevices can, at any one time, be in one of three states:

-   -   A light emission state,    -   A light detection state,    -   An idle state,        as can be advantageous for various operating strategies.

The state transitions of each inorganic LED, OLED, or related device inthe array of color inorganic LEDs, OLEDs, OLETs, or related devicesamong the above states can be coordinated in a wide variety of ways toafford various multiplexing, signal distribution, and signal gatheringschemes as can be advantageous. An exemplary general frameworkunderlying such an arrangement is described in pending U.S. patentapplication Ser. No. 12/419,229 (priority date Jan. 27, 1999), pendingU.S. patent application Ser. No. 12/471,275 (priority date May 25,2008), and in pending U.S. patent application Ser. Nos. U.S. 13/072,588and U.S. 61/517,454.

Accordingly, in an embodiment various materials, physical processes,structures, and fabrication techniques used in creating an array ofcolor inorganic LEDs, OLEDs, OLETs, or related devices and associatedco-located electronics (such as FETs, resistors, and capacitors) can beused as-is, adapted, and/or optimized so that the array of colorinorganic LEDs, OLEDs, OLETs, or related devices to work well as both acolor image display and color light-field sensor. An exemplary generalframework underlying such an arrangement is described in pending U.S.patent application 12/419,229 (priority date Jan. 27, 1999), pendingU.S. patent application 12/471,275 (priority date May 25, 2008), and inpending U.S. patent application Ser. Nos. U.S. 13/072,588 and U.S.61/517,454.

Physical Structures for Combination Light-field Sensor and Display

The capabilities described thus far can be combined with systems andtechniques to be described later in a variety of physical configurationsand implementations. A number of example physical configurations andimplementations are described here which provide various enablements andadvantages to various embodiments and implementations of the inventionand their applications. Many variations and alternatives are possibleand are accordingly anticipated by the invention, and the examplephysical configurations and implementations are in no way limiting ofthe invention.

When implementing lensless imaging cameras as taught in U.S. PatentApplication Ser. Nos. 12/419,229 (priority date Jan. 27, 1999),12/471,275 (priority date May 25, 2008), U.S. 13/072,588, and U.S.61/517,454, an optical vignetting arrangement is needed for thelight-sensing LEDs. This can be implemented in various ways. In oneexample approach, LEDs in the LED array themselves can be structured sothat light-sensing elements (including photodiodes or LEDs used inlight-sensing modes) are partially enveloped in a well comprising wallsformed in associating with at least light-emitting LEDs in the LEDarray. In a variation of this the light emitting LEDs can also be usedin a light sensing mode, in implementing a tactile user interfacearrangement as enabled by the present invention. In another variation,other types of light sensing elements (for example, photodiodes) can inimplementing an optical tactile user interface arrangement as enabled bythe present invention. FIG. 30 depicts an exemplary embodimentcomprising an LED array preceded by a vignetting arrangement as isuseful for implementing a lensless imaging camera as taught in U.S.patent application Ser. Nos. 12/419,229 (priority date Jan. 27, 1999),12/471,275 (priority date May 25, 2008), U.S. 13/072,588, and U.S.61/517,454. In another approach to be described shortly, an LCDotherwise used for display can be used to create vignetting apertures.The invention provides for inclusion of coordinated multiplexing orother coordinated between the LED array and LCD as needed oradvantageous. In another approach, a vignetting arrangement is createdas a separate structure and overlaid atop the LED array. Other relatedarrangements and variations are possible and are anticipated by theinvention. The output of the light-field sensor can also oralternatively be used to implement a tactile user interface or proximatehand gesture user interface as described later in the detaileddescription.

Leaving the vignetting considerations involved in lensless imaging,attention is now directed to arrangements wherein a transparent LEDarray, for example implemented with arrays of transparent OLEDsinterconnected with transparent conductors, is overlaid atop an LCDdisplay. The transparent conductors can be comprised of materials suchas indium tin oxide, fluorine-doped tin oxide (“FTO”), doped zinc oxide,organic polymers, carbon nanotubes, graphene ribbons, etc. FIG. 31depicts an exemplary implementation comprising a transparent OLED arrayoverlaid upon an LCD visual display, which is in turn overlaid on a(typically) LED backlight used to create and direct light though the LCDvisual display from behind. Such an arrangement may be used to implementan optical tactile user interface arrangement as enabled by the presentinvention. Other related arrangements and variations are possible andare anticipated by the invention. The invention provides for inclusionof coordinated multiplexing or other coordinated between the OLED arrayand LCD as needed or advantageous. It is noted in one embodiment the LCDand LED array can be fabricated on the same substrate, the first arraylayered atop the second (or vice versa) while in another embodiment thetwo LED arrays can be fabricated separately and later assembled togetherto form layered structure. Other related arrangements and variations arepossible and are anticipated by the invention.

FIG. 32 depicts an exemplary implementation comprising a first(transparent OLED) LED array used for at least optical sensing overlaidupon a second LED array used for at least visual display. In anembodiment, the second LED array can be an OLED array. In an embodiment,either or both of the LED arrays can comprise photodiodes. Such anarrangement can be employed to allow the first array to be optimized forone or more purposes, at least one being sensing, and the second LEDarray to be optimized for one or more purposes, at least one beingvisual display. Such an arrangement may be used to implement an opticaltactile user interface arrangement as enabled by the present invention.In one approach the second LED array is used for both visual display andtactile user interface illumination light and the first (transparentOLED) LED array is used for tactile user interface light sensing. Inanother approach, the first (transparent OLED) LED array is used forboth tactile user interface illumination light and light sensing, whilethe second LED array is used for visual display. In an embodiment, thesecond LED array is used for visual display and further comprisesvignetting structures (as described above) and serves as a light-fieldsensor to enable the implementation of a lensless imaging camera. Suchan arrangement can be used to implement a light-field sensor and alensless imaging camera as described earlier. Other related arrangementsand variations are possible and are anticipated by the invention. Theinvention provides for inclusion of coordinated multiplexing or othercoordinated between the first LED array and second LED array as neededor advantageous. It is noted in one embodiment the two LED arrays can befabricated on the same substrate, the first array layered atop thesecond (or vice versa) while in another embodiment the two LED arrayscan be fabricated separately and later assembled together to formlayered structure. Other related arrangements and variations arepossible and are anticipated by the invention.

FIG. 33 depicts an exemplary implementation comprising a first(transparent OLED) LED array used for at least visual display overlaidupon a second LED array used for at least optical sensing. In anembodiment, the second LED array can be an OLED array. In an embodiment,either or both of the LED arrays can comprise photodiodes. Such anarrangement can be employed to allow the first array to be optimized forto be optimized for other purposes, at least one being visual display,and the second LED array to be optimized for one or more purposes, atleast one being sensing. Such an arrangement may be used to implement anoptical tactile user interface arrangement as enabled by the presentinvention. In one approach the first LED array is used for both visualdisplay and tactile user interface illumination light and the second(transparent OLED) LED array is used for tactile user interface lightsensing. In another approach, the second (transparent OLED) LED array isused for both tactile user interface illumination light and lightsensing, while the first LED array is used for visual display. In anembodiment, the second LED array comprises vignetting structures (asdescribed above) and serves as a light-field sensor to enable theimplementation of a lensless imaging camera. Other related arrangementsand variations are possible and are anticipated by the invention. Theinvention provides for inclusion of coordinated multiplexing or othercoordinated between the first LED array and second LED array as neededor advantageous. It is noted in one embodiment the two LED arrays can befabricated on the same substrate, the first array layered atop thesecond (or vice versa) while in another embodiment the two LED arrayscan be fabricated separately and later assembled together to formlayered structure. Other related arrangements and variations arepossible and are anticipated by the invention.

FIG. 34 depicts an exemplary implementation comprising an LCD display,used for at least visual display and vignette formation, overlaid upon asecond LED array, used for at least backlighting of the LCD and opticalsensing. In an embodiment, the LED array can be an OLED array. In anembodiment, the LED array can comprise also photodiodes. Such anarrangement can be used to implement an optical tactile user interfacearrangement as enabled by the present invention. Such an arrangement canbe used to implement a light-field sensor and a lensless imaging cameraas described earlier. The invention provides for inclusion ofcoordinated multiplexing or other coordinated between the LCD and LEDarray as needed or advantageous. It is noted in one embodiment the LCDand LED array can be fabricated on the same substrate, the first arraylayered atop the second (or vice versa) while in another embodiment thetwo LED arrays can be fabricated separately and later assembled togetherto form layered structure. Other related arrangements and variations arepossible and are anticipated by the invention.

Use of a LED Array as “Multi-Touch” Tactile Sensor Array

Multitouch sensors on cellphones, smartphones, PDAs, tablet computers,and other such devices typically utilize a capacitive matrix proximitysensor. FIG. 35 an exemplary arrangement employed in contemporarycellphones, smartphones, PDAs, tablet computers, and other portabledevices wherein a transparent capacitive matrix proximity sensor isoverlaid over an LCD display, which is in turn overlaid on a (typicallyLED) backlight used to create and direct light though the LCD displayfrom behind. Each of the capacitive matrix and the LCD have considerableassociated electronic circuitry and software associated with them.

FIG. 36 depicts an exemplary modification of the arrangement depicted inFIG. 35 wherein the LCD display and backlight are replaced with an OLEDarray used as a visual display. Such an arrangement has started to beincorporated in recent contemporary cellphone, smartphone, PDA, tabletcomputers, and other portable device products by several manufacturers.Note the considerable reduction in optoelectronic, electronic, andprocessor components over the arrangement depicted in FIG. 35. This isone of the motivations for using OLED displays in these emerging productimplementations.

FIG. 37 depicts an exemplary arrangement provided for by the inventioncomprising only a LED array. The LEDs in the LED array may be OLEDs orinorganic LEDs. Such an arrangement can be used as a tactile userinterface, or as a combined a visual display and tactile user interface,as will be described next. Note the considerable reduction inoptoelectronic, electronic, and processor components over the both thearrangement depicted in FIG. 35 and the arrangement depicted in FIG. 36.This is one of the advantages of many embodiments of the presentinvention.

Arrays of inorganic-LEDs have been used to create a tactile proximitysensor array as taught by Han in U.S. Pat. No. 7,598,949 and as depictedin the video available at http://cs.nyu.edutabout.jhan/ledtouch/index.html). FIG. 38 depicts a representativeexemplary arrangement wherein light emitted by neighboring LEDs isreflected from a finger (or other object) back to an LED acting as alight sensor.

In its most primitive form, such LED-array tactile proximity arrayimplementations need to be operated in a darkened environment (as seenin the video available at http://cs.nyu.edutabout.jhan/ledtouch/index.html). The invention provides for additionalsystems and methods for not requiring darkness in the user environmentin order to operate the LED array as a tactile proximity sensor.

In one approach, potential interference from ambient light in thesurrounding user environment can be limited by using an opaque pliableand/or elastically deformable surface covering the LED array that isappropriately reflective (directionally, amorphously, etc. as may beadvantageous in a particular design) on the side facing the LED array.Such a system and method can be readily implemented in a wide variety ofways as is clear to one skilled in the art.

In another approach, potential interference from ambient light in thesurrounding user environment can be limited by employing amplitude,phase, or pulse width modulated circuitry and/or software to control thelight emission and receiving process. For example, in an implementationthe LED array can be configured to emit modulated light that ismodulated at a particular carrier frequency and/or with a particulartime-variational waveform and respond to only modulated light signalcomponents extracted from the received light signals comprising thatsame carrier frequency or time-variational waveform. Such a system andmethod can be readily implemented in a wide variety of ways as is clearto one skilled in the art.

The light measurements used for implementing a tactile user interfacecan be from unvignetted LEDs, unvignetted photodiodes, vignetted LEDs,vignetted photodiodes, or combinations of two or more of these.

Separate Sensing and Display Elements in an LED Array

In one embodiment provided for by the invention, some LEDs in an arrayof LEDs are used as photodetectors while other elements in the array areused as light emitters. The light emitter LEDs can be used for displaypurposes and also for illuminating a finger (or other object)sufficiently near the display. FIG. 39 depicts an exemplary arrangementwherein a particular LED designated to act as a light sensor issurrounded by immediately-neighboring element designated to emit lightto illuminate the finger for example as depicted in FIG. 38. Otherarrangements of illuminating and sensing LEDs are of course possible andare anticipated by the invention.

It is also noted that by dedicating functions to specific LEDs as lightemitters and other elements as light sensors, it is possible to optimizethe function of each element for its particular role. For example, in anexample embodiment the elements used as light sensors can be optimizedphotodiodes. In another example embodiment, the elements used as lightsensors can be the same type of LED used as light emitters. In yetanother example embodiment, the elements used as light sensors can beLEDs that are slightly modified versions the of type of LED used aslight emitters.

In an example embodiment, the arrangement described above can beimplemented only as a user interface. In an example implementation, theLED array can be implemented as a transparent OLED array that can beoverlaid atop another display element such as an LCD or another LEDarray. In an implementation, LEDs providing user interface illuminationprovide light that is modulated at a particular carrier frequency and/orwith a particular time-variational waveform as described earlier.

In an alternative example embodiment, the arrangement described abovecan serve as both a display and a tactile user interface. In an exampleimplementation, the light emitting LEDs in the array are time-divisionmultiplexed between visual display functions and user interfaceillumination functions. In another example implementation, some lightemitting LEDs in the array are used for visual display functions whilelight emitting LEDs in the array are used for user interfaceillumination functions. In an implementation, LEDs providing userinterface illumination provide modulated illumination light that ismodulated at a particular carrier frequency and/or with a particulartime-variational waveform. In yet another implementation approach, themodulated illumination light is combined with the visual display lightby combining a modulated illumination light signal with a visual displaylight signal presented to each of a plurality of LEDs within the in theLED array. Such a plurality of LEDs can comprise a subset of the LEDarray or can comprise the entire LED array.

FIG. 40 depicts an exemplary arrangement wherein a particular LEDdesignated to act as a light sensor is surrounded byimmediately-neighboring LEDs designated to serve as a “guard area,” forexample not emitting light, these in turn surrounded byimmediately-neighboring LEDs designated to emit light used to illuminatethe finger for example as depicted in FIG. 38. Such an arrangement canbe implemented in various physical and multiplexed ways as advantageousto various applications or usage environments.

In an embodiment, the illumination light used for tactile user interfacepurposes can comprise an invisible wavelength, for example infrared orultraviolet.

Sequenced Sensing and Display Modes for LEDs in an LED Array

In another embodiment provided for by the invention, each LED in anarray of LEDs can be used as a photodetector as well as a light emitterwherein each individual LED can either transmit or receive informationat a given instant. In an embodiment, each LED in a plurality of LEDs inthe LED array can sequentially be selected to be in a receiving modewhile others adjacent or near to it are placed in a light emitting mode.Such a plurality of LEDs can comprise a subset of the LED array or cancomprise the entire LED array. A particular LED in receiving mode canpick up reflected light from the finger, provided by said neighboringilluminating-mode LEDs. In such an approach, local illumination andsensing arrangements such as that depicted FIG. 39 (or variationsanticipated by the invention) can be selectively implemented in ascanning and multiplexing arrangement.

FIG. 40 depicts an exemplary arrangement wherein a particular LEDdesignated to act as a light sensor is surrounded byimmediately-neighboring LEDs designated to serve as a “guard” area, forexample not emitting light, these in turn surrounded byimmediately-neighboring LEDs designated to emit light used to illuminatethe finger for example as depicted in FIG. 38. Such an arrangement canbe implemented in various multiplexed ways as advantageous to variousapplications or usage environments.

In an alternative example embodiment, the arrangement described abovecan serve as both a display and a tactile user interface. In an exampleimplementation, the light emitting LEDs in the array are time-divisionmultiplexed between visual display functions and user interfaceillumination functions. In another example implementation, some lightemitting LEDs in the array are used for visual display functions whilelight emitting LEDs in the array are used for user interfaceillumination functions. In an implementation, LEDs providing userinterface illumination provide modulated illumination light that ismodulated at a particular carrier frequency and/or with a particulartime-variational waveform. In yet another implementation approach, themodulated illumination light is combined with the visual display lightby combining a modulated illumination light signal with a visual displaylight signal presented to each of a plurality of LEDs within the in theLED array. Such a plurality of LEDs can comprise a subset of the LEDarray or can comprise the entire LED array.

Use of a LED Array in Implementing a Lensless Imaging Camera

In an embodiment, various materials, physical processes, structures, andfabrication techniques used in creating an array of color inorganicLEDs, OLEDs, OLETs, or related devices and associated co-locatedelectronics (such as FETs, resistors, and capacitors) can be used as-is,adapted, and/or optimized so as to create an array of color inorganicLEDs, OLEDs, OLETs, or related devices that is well-suited as foroperation as a color lensless imaging camera according to the generalframework described in pending U.S. patent application Ser. No.12/419,229 (priority date Jan. 27, 1999), pending U.S. patentapplication Ser. No. 12/471,275 (priority date May 25, 2008), and inpending U.S. patent application Ser. Nos. U.S. 13/072,588 and U.S.61/517,454.

Co-pending patent application Ser. Nos. U.S. 12/471,275 (priority dateJan. 27, 1999), U.S. 12/419,229, U.S. 13/072,588, and U.S. 61/517,454teach, among other things, the use of an LED array, outfitted withmicro-optical arrangements that create optical occulting/vignetting andinterfaced with a new type of “image formation” signal processing, as a(monochrome or color) camera for image or video applications.

Image formation is performed without a conventional large shared lensand associated separation distance between lens and image sensor,resulting in a “lensless camera.”

Further, the similarities between the spectral detection band and thespectral emission bands of each of a plurality of types of colored-lightLED may be used to create a color light-field sensor from a color LEDarray display such as that currently employed in “LED TV” products,large outdoor color-image LED displays (as seen in advertising signs andsports stadiums), and recently (in the form of OLED displays) in SamsungSmartphones.

In an embodiment, the various materials, physical processes, structures,and fabrication techniques used in creating the LED array and associatedco-located electronics (such as FETs, resistors, and capacitors) may beused to further co-optimize a high performance monochrome LED array orcolor LED array to work well as both an image display and light-fieldsensor compatible with synthetic optics image formation algorithms usingmethods, systems, and process such as those aforedescribed.

Operation as a Combination Color Lensless Imaging Camera and ColorVisual Display

The examples above employ an LED array multiplexed in betweenlight-emitting modes and light-sensing modes. The examples above employan LED array to be used in visual (image, video, GUI) display modes andlensless imaging camera modes. The invention provides for these to beintegrated together into a common system, leveraging one or more of ashared electronics infrastructure, shared processor infrastructure, andshared algorithmic infrastructure.

In an embodiment, an array of color inorganic LEDs, OLEDs, OLETs, orrelated devices, together with associated signal processing aspects ofthe invention, can be adapted to function as both a color image displayand color light-field sensor compatible with synthetic optics imageformation algorithms using methods, systems, and process such as thosedescribed in pending U.S. patent application Ser. No. 12/419,229(priority date Jan. 27, 1999), pending U.S. patent application Ser. No.12/471,275 (priority date May 25, 2008), and in pending U.S. patentapplication Ser. Nos. U.S. 13/072,588 and U.S. 61/517,454.

Either of these arrangements allows for a common processor to be usedfor display and camera functionalities. The result dramaticallydecreases the component count and system complexity for contemporary andfuture mobile devices such as cellphones, smartphones, PDAs tabletcomputers, and other such devices.

Use of a LED Array as an Integrated Camera and a Display Simultaneously

The invention further provides for a common LED array multiplexed inbetween light-emitting display modes and light-sensing lensless imagingcamera modes. Here the LED array to be time multiplexed between “imagecapture” and “image display” modes so as to operate as an integratedcamera/display. The invention provides for these to be integratedtogether into a common system, leveraging one or more of a sharedelectronics infrastructure, shared processor infrastructure, and sharedalgorithmic infrastructure.

The integrated camera/display operation removes the need for ascreen-direction camera and interface electronics in mobile devices. Theintegrated camera/display operation can also improve eye contact inmobile devices.

Employing these constructions, the invention provides for an LED arrayimage display, used in place of a LCD image display, to serve as atime-multiplexed array of light emitter and light detector elements. Theresulting system does not require an interleaving or stacking offunctionally-differentiated (with respect to light detection and lightemission) elements. This is particularly advantageous as there is a vastsimplification in manufacturing and in fact close or precise alignmentwith current LED array image display manufacturing techniques andexisting LED array image display products.

Either of these arrangements allows for a common processor to be usedfor display and camera functionalities. The result dramaticallydecreases the component count and system complexity for contemporary andfuture mobile devices such as cellphones, smartphones, PDAs, tabletcomputers, and other such devices.

Use of a LED Array in Implementing a (Tactile User Interface) TouchScreen Sensor

In an embodiment, an array of color inorganic LEDs, OLEDs, OLETs, orrelated devices, together with associated signal processing aspects ofthe invention, can be used to implement a tactile (touch-based) userinterface sensor.

Use of a LED Array in Implementing a Combination (Tactile UserInterface) Touch Screen Sensor and Color Visual Image Display

In an embodiment, an array of color inorganic LEDs, OLEDs, OLETs, orrelated devices, together with associated signal processing aspects ofthe invention, can be adapted to function as both a color image visualdisplay and light-field sensor which can be used to implement a tactileuser interface.

The integrated tactile user interface sensor capability can remove theneed for a tactile user interface sensor (such as a capacitive matrixproximity sensor) and associated components.

Either of these arrangements allows for a common processor to be usedfor display and camera functionalities. The result dramaticallydecreases the component count and system complexity for contemporary andfuture mobile devices such as cellphones, smartphones, PDAs, tabletcomputers, and other such devices.

Use of a LED Array as an Integrated Camera, Display, and (Tactile UserInterface) Touch Screen

In an approach to the invention, the LED array is multiplexed amongdisplay, camera, and tactile user interface sensor modalities. In anexemplary associated embodiment of the invention, an LED array can beused as a display, camera, and touch-based user interface sensor.

The integrated camera/display operation removes the need for ascreen-direction camera and interface electronics in mobile devices. Theintegrated camera/display operation can also improve eye contact inmobile devices.

The integrated tactile user interface sensor capability can remove theneed for a tactile user interface sensor (such as a capacitive matrixproximity sensor) and associated components.

Either of these arrangements allows for a common processor to be usedfor display, camera, and tactile user interface sensor functionalities.The result dramatically decreases the component count and systemcomplexity for contemporary and future mobile devices such assmartphones PDAs, tablet computers, and other such devices.

Use of a LED Array in Implementing a Proximate Gesture User InterfaceSensor

In an embodiment, an array of color inorganic LEDs, OLEDs, OLETs, orrelated devices, together with associated signal processing aspects ofthe invention, can be adapted to function as both an image display andlight-field sensor which can be used to implement a proximate (handimage) gesture user interface sensor as, for example, replacing thephotodiode sensor arrangement used in the M.I.T. BI-DI user interface.In one approach, the M.I.T. BI-DI photodiode sensor arrangement behindthe LCD array can be replaced with an array of inorganic LEDs, OLEDs, orrelated devices as provided for in the current invention. In anotherapproach, the M.I.T. BI-DI photodiode sensor arrangement behind the LCDarray, the LCD itself, and the associated backlight can be replaced withan array of inorganic LEDs, OLEDs, or related devices as provided for inthe current invention.

The integrated proximate (hand image) gesture user interface capabilitycan remove the need for a tactile user interface sensor (such as acapacitive matrix proximity sensor) and associated components.

Either of these arrangements allows for a common processor to be usedfor display, camera, and tactile user interface sensor functionalities.The result dramatically decreases the component count and systemcomplexity for contemporary and future mobile devices such ascellphones, smartphones, PDAs, tablet computers, and other such devices.

Use in Implementing a Combination Proximate Gesture User Interface andVisual Image Display

In an embodiment, an array of color inorganic LEDs, OLEDs, OLETs, orrelated devices, together with associated signal processing aspects ofthe invention, can be adapted to function as both a visual image displayand light-field sensor which can be used to implement a proximategesture user interface sensor as, for example, replacing the photodiodesensor and LCD display arrangement used in the M.I.T. BI-DI userinterface. In an embodiment an array of inorganic LEDs, OLEDs, orrelated devices can be adapted to function as both an image display andlight-field sensor which can be used to implement a tactile userinterface sensor, and also as a visual display.

The integrated proximate (hand image) gesture user interface capabilitycan remove the need for a tactile user interface sensor (such as acapacitive matrix proximity sensor) and associated components.

Either of these arrangements allows for a common processor to be usedfor display, camera, and tactile user interface sensor functionalities.The result dramatically decreases the component count and systemcomplexity for contemporary and future mobile devices such ascellphones, smartphones, PDAs, tablet computers, and other such devices.

Use in Implementing a Combination Proximate Gesture User InterfaceSensor, Lensless Imaging Camera, and Visual Image Display

In an embodiment, an array of color inorganic LEDs, OLEDs, OLETs, orrelated devices, together with associated signal processing aspects ofthe invention, can be adapted to function as both an image display andlight-field sensor which can be used to implement a proximate gestureuser interface sensor, lensless imaging camera, and visual imagedisplay.

The integrated camera/display operation removes the need for ascreen-direction camera and interface electronics in mobile devices. Theintegrated camera/display operation can also improve eye contact inmobile devices.

The integrated proximate (hand image) gesture user interface capabilitycan remove the need for a tactile user interface sensor (such as acapacitive matrix proximity sensor) and associated components.

Either of these arrangements allows for a common processor to be usedfor display, camera, and tactile user interface sensor functionalities.The result dramatically decreases the component count and systemcomplexity for contemporary and future mobile devices such ascellphones, smartphones, PDAs, tablet computers, and other such devices.

Use in Implementing a Combination Proximate Gesture User Interface,Tactile User Interface (Touchscreen), and Visual Image Display

In an embodiment, an array of color inorganic LEDs, OLEDs, OLETs, orrelated devices, together with associated signal processing aspects ofthe invention, can be adapted to function as both an image display andlight-field sensor which can be used to implement a proximate gestureuser interface, tactile user interface (touchscreen), and visual imagedisplay.

Further, the integrated combination of a tactile user interface (touchscreen) capability and a proximate (hand image) gesture user interfacecapability can provide a more flexible, sophisticated, and higheraccuracy user interface.

Further, as the integrated tactile user interface (touch screen) and aproximate (hand image) gesture user interface capabilities employ thelight filed sensor modalities for the LED array, this approach canremove the need for a tactile user interface sensor (such as acapacitive matrix proximity sensor) and associated components.

Either of these arrangements allows for a common processor to be usedfor display, camera, proximate gesture user interface, and tactile userinterface (touch screen) functionalities. The result dramaticallydecreases the component count and system complexity for contemporary andfuture mobile devices such as cellphones, smartphones, PDAs, tabletcomputers, and other such devices.

Use in Implementing a Combination Proximate Gesture User Interface,Tactile User Interface (Touchscreen), Lensless Imaging Camera, andVisual Image Display

In an embodiment, an array of color inorganic LEDs, OLEDs, OLETs, orrelated devices, together with associated signal processing aspects ofthe invention, can be adapted to function as both an image display andlight-field sensor which can be used to implement a proximate gestureuser interface, tactile user interface (touchscreen), lensless imagingcamera, and visual image display.

The integrated camera/display operation removes the need for ascreen-direction camera and interface electronics in mobile devices. Theintegrated camera/display operation can also improve eye contact inmobile devices.

Further, the integrated combination of a tactile user interface (touchscreen) capability and a proximate (hand image) gesture user interfacecapability can provide a more flexible, sophisticated, and higheraccuracy user interface.

Either of these arrangements allows for a common processor to be usedfor display, camera, proximate gesture user interface, and tactile userinterface (touch screen) functionalities. The result dramaticallydecreases the component count and system complexity for contemporary andfuture mobile devices such as cellphones, smartphones, PDAs, tabletcomputers, and other such devices.

System Architecture Advantages and Consolidation Opportunities forMobile Devices

The arrangements described above allow for a common processor to be usedfor display and camera functionalities. The result dramaticallydecreases the component count, system hardware complexity, andinter-chip communications complexity for contemporary and future mobiledevices such as cellphones, smartphones, PDAs, tablet computers, andother such devices.

FIG. 41 depicts an exemplary reference arrangement comprised by mobiledevices such as cellphones, smartphones, PDAs, tablet computers, andother such devices.

FIG. 42 depicts an exemplary variation of the exemplary referencearrangement of FIG. 41 wherein an LED array replaces the display,camera, and touch sensor and is interfaced by a common processor thatreplaces associated support hardware. In an embodiment, the commonprocessor is a Graphics Processing Unit (“GPU”) or comprises a GPUarchitecture.

FIG. 43 depicts an exemplary variation of the exemplary referencearrangement of FIG. 42 wherein the common processor associated with theLED array further executes at least some touch-based user interfacesoftware.

FIG. 44 depicts an exemplary variation of the exemplary referencearrangement of FIG. 42 wherein the common processor associated with theLED array further executes all touch-based user interface software.

Advantageous Use of Supplemental Integrated Capacitive Sensors

In an embodiment, a capacitive sensor may be used to supplement theoptical tactile sensing capabilities describe thus far. FIG. 45 depictsan exemplary embodiment comprising an LED array preceded by amicrooptics vignetting array and a capacitive matrix sensor. FIG. 46depicts an exemplary embodiment comprising an LED array preceded by amicrooptics vignetting array that is physically and electricallyconfigured to also function as a capacitive matrix sensor. This can beaccomplished by making at least some portions of the microopticsvignetting array electrically conductive.

Closing

While the invention has been described in detail with reference todisclosed embodiments, various modifications within the scope of theinvention will be apparent to those of ordinary skill in thistechnological field. It is to be appreciated that features describedwith respect to one embodiment typically can be applied to otherembodiments.

The invention can be embodied in other specific forms without departingfrom the spirit or essential characteristics thereof. The presentembodiments are therefore to be considered in all respects asillustrative and not restrictive, the scope of the invention beingindicated by the appended claims rather than by the foregoingdescription, and all changes which come within the meaning and range ofequivalency of the claims are therefore intended to be embraced therein.Therefore, the invention properly is to be construed with reference tothe claims.

Although exemplary embodiments have been provided in detail, it shouldbe understood that various changes, substitutions and alternations couldbe made thereto without departing from spirit and scope of the disclosedsubject matter as defined by the appended claims. Variations describedfor exemplary embodiments may be realized in any combination desirablefor each particular application. Thus particular limitations, and/orembodiment enhancements described herein, which may have particularadvantages to a particular application, need not be used for allapplications. Also, not all limitations need be implemented in methods,systems, and/or apparatuses including one or more concepts describedwith relation to the provided exemplary embodiments.

What is claimed is:
 1. A system for implementing a function of a visualdisplay, light-field sensor, and a user interface for operation by auser hand, the system comprising: at least one processor, the at leastone processor having an electrical interface and for executing at leastone software algorithm; a liquid crystal display (LCD) configured to bein communication with the at least one processor; a light emitting diode(LED) array having light-sensing capabilities, the LED array arrangedfor providing backlighting for the LCD and light sensing through theLCD, the LED array in communication with the at least one processor;wherein the LCD is used for at least visual display and vignetteformation acting on an incoming light-field, wherein light sensingportions of the LED array perform a light detection function at leastfor an interval of time, a result of the light detection functioncomprising incoming light-field measurement information communicated tothe at least one processor, and wherein the least one processorcalculates an image from the light-field measurement informationresponsive to the incoming light-field as measured through thevignetting provided by the LCD.
 2. The system of claim 1, wherein theLED array comprises at least one organic light-emitting diode (OLED). 3.The system of claim 2, wherein each OLED is operated in at least two ofa light emission state, a light detection state, and an idle state. 4.The system of claim 1, wherein the LED array comprises light sensingelements.
 5. The system of claim 4, wherein the light sensing elementscomprise photodiodes.
 6. The system of claim 1, wherein the LED arraycomprises two LED arrays that are fabricated separately and assembledtogether to form a layered structure.
 7. The system of claim 1, whereinthe light sensing portions of the LED array are transparent.
 8. Thesystem of claim 1, wherein the system is configured to operate as bothan image display and a light-field sensor.
 9. The system of claim 1,wherein the light-field measurement information is used to createlight-field sensor output information.
 10. The system of claim 9,wherein the light-field sensor output information is used to implement aproximity hand gesture user interface.
 11. The system of claim 9,wherein the light-field sensor output information is used to implement atactile user interface.
 12. The system of claim 1, where the system isconfigured to implement a touch screen user interface.
 13. The system ofclaim 1, wherein the system is configured to implement a proximate imageuser interface sensor.
 14. The system of claim 1, where the system isconfigured to implement a lensless imaging camera calculating the imagefrom the incoming light-field as measured through the vignettingprovided by the LCD.
 15. The system of claim 1, where the system iscomprised within a mobile phone.
 16. The system of claim 1, where thesystem is comprised within a Personal Digital Assistant (“PDA”).
 17. Thesystem of claim 1, where the system is comprised within a tabletcomputer.
 18. The system of claim 1, wherein the system is configured tooperate simultaneously as an imaging camera and as a display.
 19. Thesystem of claim 18, wherein the system is further configured to provideeye-contact in video conferencing applications.
 20. The system of claim1, wherein the system is configured to operate simultaneously as a videocamera and a display.